Assistant Professor | School of Information Science
You have a maximum of 10 pages for your typed, double-spaced responses; be accurate and concise (like a quantitative researcher).
Why is quantitative research important to the communication discipline?
Among other reasons…
What is internal validity? How is it evaluated? What are the potential major threats to internal validity and what can you do to reduce them?
Internal validity depends on the strength or soundness of the design and influences whether one can conclude that the independent variable or intervention caused the dependent variable to change
We evaluate it based on…
What is external validity? How does a theoretical population differ from a sampling frame? An accessible sample? An actual sample? How do you determine the usefulness of a sampling frame?
External validity is the extent to which samples, settings, and variables can be generalized beyond the study.
Theoretical population: All the participants of theoretical interest to the research and to which he or she would like to generalize
Sampling frame: The group of participants you actually have access to, perhaps through a list or directory
Selected sample: Smaller group selected from the larger accessible population by the researcher and asked to participate in the study.
Actual sample: Participants that complete the study and whose data are actually used in the analysis and in the report of the results.
The sampling frame represents an exhaustive list of the participants that a researcher could realistically access for a study.
Why is measurement important to quantitative research? How does one go about building a measure that is both reliable and valid? What are the critical steps?
Measurement is the assignment of numbers or symbols to the different levels or values of variables according to rules.
In social science, we often want to assess ideas that we cannot directly observe.
Effective measurement aligns conceptualization with operationalization and enhances internal validity.
Measurement also influences which statistics we use to draw conclusions and generalizations.
“Research results are no more valid than the measures used to collect the data” (Levine, 2005, p. 335).
“Poor measurement misrepresents the social phenomena being studied, and a lack of construct validity could lend support to inaccurate theories and distort accurate theories” (Bowman & Goodboy, 2020, p. 232).
Stay tuned!!
Reliability: Is the measuring performing consistently across respondents?
Validity: Whether the scores provide evidence for the use of a measure in a specific setting.
Reliability is necessary for validity, but one can have consistent data that is not valid.
Scale development is useful for capturing not directly observable concepts
What makes a good quant study? What criteria would you use to distinguish a good study from a poor study? What are some of the limitations of quantitative work?
What is the purpose of experimental research? What are its strengths and weaknesses?
What is the purpose of survey research? What are its strengths and weaknesses?
To make trusthworhty generalizations from surveys…
What are the characteristics of a good research question / hypothesis? Can you pose at least one for an idea that interests you?
How would you design an experimental or survey-based approach to answering this RQ or hypothesis?
How would your design address:
Let’s say your group wants to study men and womens’ reactions to violent crime shows on national TV.
With each:
We’re finally talking about targets!
Study quality depends on the consistency and accuracy of measurement instruments.
Think about what it means to be a reliable person.
Reflects the consistency of a series of measurements
Are people responding to a measure consistently over time?
If our outcome measure does not provide reliable data, then we cannot accurately assess the results of our study.
How do we know that a score is due to the intervention or due to some other, unsystematic factor?
Reliability is a coefficient
The ratio of the variance of true scores to the variance of observed scores
The higher the reliability of the data, the closer the true scores will be to observed scores
Reliability will be displayed as a correlation
This indicates the strength of the relationship between two variables
A strong positive relationship indicates that people who score high on one test also will score high on a second test. To say that scores from a measure are reliable, one usually would expect a coefficient between +.7 and +1.0.
Reliabilities will not be negative (or something is wrong)
\(\alpha\)
\(\omega\)
An instrument of support was used to measure perceived support from coworkers in a mental health institution. Participants responded to four items on a seven-point Likert-like scale. Cronbach’s alpha for the (support) scale was .79. What does this mean?
If we have less reliable measures, we are less confident that our participants’ observed scores are close to their true scores.
Any score that we obtain from an individual on an instrument
Observed score = True Score + Error
We cannot know the true score, but we can estimate where it might fall
A range of scores (i.e., confidence interval) within which should lie a performer’s true score.
Most of the time, we want to be 95% sure that we capture the true score (2 standard deviations)
Lets say we are measuring satisfaction
\(\sigma^2\) = 15 \(\alpha\) = .92 SEM = 4.24
Someone completes the combined measures and scores a 110 total
If the instrument has 20 Likert questions ranging from 1 to 7, possible values are 20 to 140.
The z score for any score that is 2 standard deviations away is 1.96
1.96 [Z score] * 4.24 [SEM] = 8.32
We can conclude that our true score falls within the 95 percent confidence interval of 110 ± 8.32 or between 101.68 and 118.32
If the test were given to the same person a large number of times, 95 percent of the confidence intervals would contain the true score.
Lets say we are measuring satisfaction
\(\sigma^2\) = 15 \(\alpha\) = .65 SEM = 8.87
Someone completes the combined measures and scores a 110 total
If the instrument has 20 Likert questions ranging from 1 to 7, possible values are 20 to 140.
The z score for any score that is 2 standard deviations away is 1.96
1.96 [Z score] * 8.87 [SEM] = 17.39
We can conclude that our true score falls within the 95 percent confidence interval of 110 ± 17.39 or between 92.61 and 127.61
We are less precise in our understanding of how participants are responding
We’re finally talking about targets!
Study quality depends on the consistency and accuracy of measurement instruments.
Concerned with establishing evidence for the use of a particular measure or instrument in a particular setting with a particular population for a specific purpose.
accumulating evidence to provide a sound scientific basis for the proposed score interpretations.
Are we hitting the target?
Reliability is necessary for validity, but one can have consistent data that is not valid.
Suppose I used measurements of your dart throws to indicate your ability to pass this class.
Validity is tough to establish and takes multiple studies
One type of evidence is insufficient for validity
Often must demonstrate performance of measure in relation to other, similar measures
There is no statistic for validity
Predictive: Do scores on grit predict graduation from West Point?
Concurrent: Does grit relate to amount of time studying for a test?
Convergent: Is grit related to a measure of a theoretically similar variable (e.g., resilience)
Divergent: Is grit different from a theoretically similar variable (e.g., different from coping)
The goal should be to build evidence that a measure performs in ways one would expect.
In general, it is advisable to select instruments that have been used in other studies if they have been shown to produce reliable and valid data with the types of participants and for the purpose that you have in mind.
For your projects…
General Guidelines
Fundamental goal at this stage is to sample systematically all content that is potentially relevant to the target construct.
You will drop weak items but cannot add them back.
Recommend Likert scale but you can use semantic differential if you really want.